JQ

the JSON Processor

Author: David Montaño

Agenda

Introduction

  • Synopsis
  • Filters
  • Invocation Options

Basic Usage

  • Basic Filters
  • Types and Values
  • Operators and Functions
  • Conditionals and Comparisons
  • Regular Expressions

Advanced Usage

  • Advanced Features
  • Streaming
  • Assignement
  • Modules

Introduction

Synopsis

JQ can transform JSON documents by:

  • selecting
  • iterating
  • reducing
  • mangling

Filters

  • take an input and return an output
  • can be combined / pipe one filter into another
  • no iterations or loops: just glue filters

Invocation Options

Affect how JQ reads and writes its input and output

  • --unbuffered
  • --raw-input
  • -f / --from-file
  • --version
  • etc...(man jq)

Basic Usage

Basic Filters

  • .
  • .foo, .foo.bar
  • .foo?
  • .[string], .[2], .[10:15]
  • .[]
  • ,
  • |

Types and Values

  • numbers
  • strings
  • booleans
  • arrays
  • objects
  • If you have a filter X that produces four results, then the expression [X] will produce a single result, an array of four elements.

Operators and Functions

  • + - * / %
  • length
  • keys, keys_unsorted
  • has(key)
  • in
  • path(path_expression)
  • del(path_expression)
  • to_entries, from_entries, with_entries
  • select(boolean_expression)
  • arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars
  • map(x), map_values(x)
  • paths, paths(node_filter)

Operators and Functions

  • any, all
  • flatten
  • indices(s)
  • index(s), rindex(s)
  • inside
  • startswith(str), endwith(str)
  • combinations, combinations(n)
  • Dates: now, gmtime, mktime, strptime, ...
  • error(error message)
  • ...

Conditionals and Comparisons

  • == !=
  • if then else
  • > >= <= <
  • and or not
  • Alternative operator: //
  • try catch

Regular Expressions

Uses the Oniguruma REGEX library

  • test(val), test(regex; flags)
  • match(val), match(regex; flags)
  • capture(val), capture(regex; flags)
  • ...

Advanced Usage

  • variables
  • function definition
  • reduce
  • limit(n; exp)
  • foreach
  • recursion
  • generators and iteretors

Streaming

  • jq --stream ...
  • jq begins to work immediately
  • imply a streamed form
    • [0,[1]] is [[0],0],[[1,0],1],[[1,0]],[[1]]
      • functions
        • truncate_stream(stream_expression)
        • fromstream(stream_expression)
        • tostream

Assignement

Assignment works a little differently in jq than in most programming languages. jq doesn't distinguish between references to and copies of something - two objects or arrays are either equal or not equal, without any further notion of being "the same object" or "not the same object".

Modules

jq has a library/module system. Modules are files whose names end in .jq.

Thanks!